ON CLOSED INVARIANT SETS IN LOCAL DYNAMICS 



CINZIA BISI * 

Abstract. We investigate the dynamical behaviour of a holomorphic map on 
a /—invariant subset C of U, where / : U —> C k . We study two cases: when 
U is an open, connected and polynomially convex subset of C k and C CC U, 
closed in U, and when dU has a p.s.h. barrier at each of its points and C is not 
relatively compact in U. In the second part of the paper, we prove a Birkhoff 's 
type Theorem for holomorphic maps in several complex variables, i.e. given 
an injective holomorphic map /, defined in a neighborhood of U, with U star- 
shaped and f(U) a Runge domain, we prove the existence of a unique, forward 
invariant, maximal, compact and connected subset of U which touches dU. 



1. Introduction 

Let / : U — > be a holomorphic map. Here U is an open, connected and 
bounded (or hyperbolic) subset in C k . Since the semi-local holomorphic dynamics 
isn't well understood yet, specially when k > 2, we describe the dynamical be- 
haviour of / on a /—invariant subset C of U in two different cases: 

a) when C CC U, closed in U, and U is polynomially convex; 

b) when C isn't relatively compact in U and every point in dU has a p.s.h. 
barrier. 

When there is a recurrent component V in the interior of the polynomially convex 
hull of C in case a) or in the interior of C in case b), we prove that the dynamical 
behaviour on V is of three types: 

1. V is the basin of attraction of an attractive periodic orbit; 

2. V is a Siegel domain; 

3. if h is a limit of a subsequence of {f n } n eN, then < rank(/i) < k. 

In particular when C is a closed orbit or a countable union of closed orbits, we 
prove that C cannot have a non-empty interior with a recurrent point. This has 
been proved by Fornaess-Stensones in [BJ when U has a Lipschitz boundary; here it 
is proved in a different situation, i.e. when U is polynomially convex or with a p.s.h. 
barrier at each boundary point, then U has not necessarily Lipschitz boundary. 
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In the second part of the paper, see section [H we give a version of Birkhoff's The- 
orem which was originally stated for surface transformations / having a Lyapunov 
unstable fixed point p for / or for Under these hypotheses Birkhoff has shown, 
[3] , the existence, in each neighborhood U of p, of a compact set K + (or K_ ) which 
is positive (or negative) invariant by / and touching the boundary of U. In this 
general setting there is no forward and backward invariant compact set with this 
property. 

In the same spirit, our theorem 14.11 asserts that if / : U — » C fc is a holomorphic 
injective map of C k such that /(0) = 0, with U bounded and star-shaped and f(U) 
a Runge domain, then there exists a unique, maximal, compact, connected set K 
such that: 

1. e K cU; 

2. KndU^V): 

3. f(K) C K. 

In general, this compact set K isn't totally invariant: we will give an example, sec 
example 15. II So the several variables analogue of R. Perez-Marco's hedgehogs, [15] . 
does not hold: in the one variable case the compact is totally invariant and touches 
the boundary, [i"5] . 

2. Preliminaries 

We recall some definitions and fix our notations. 
Let K be a compact set of C fe , then the polynomially convex hull of K is defined 
as: 

K-p = {z G C fe | \p(z)\ < sup \p(Q\ Vp polynomial}. 
A compact set K is polynomially convex if K = K-p. 

Definition 2.1. An open set U in C k is polynomially convex if, for every compact 
K in U, K v CC U. 

For example, the geometrically convex open sets of C k are polynomially convex 
in C fc . The property of being polynomially convex is not invariant by biholomor- 
phisms, as Wermer showed, see Gunning's book, [11] , page 46. 
If K is polynomially convex, each holomorphic function on a neighborhood of K is 
the uniform limit on K of polynomials; in the same way if p is p.s.h. and continuous 
on U, polynomially convex open set, then it is the uniform limit on the compact 
sets of U of p.s.h. functions of C k . 

A consequence, when U is polynomially convex, is that convexity with respect to 
p.s.h. functions in U is the same as polynomial convexity. 

If K is polynomially convex and compact in U, there exists p\ p.s.h. and continuous 
on C fc , K = {pi < 0} and p\ > 1 on a neighborhood of C fc \ U. 

Definition 2.2. A domain U is Runge if each holomorphic function on U can be 
approximated by polynomials, uniformly on compact subsets of U. 

In particular any polynomially convex open set is a Runge domain, [llj . 
It is possible to construct Runge domains such that the interior of U is not equal 
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to U : for example U — {w G C fc : |to| < exp(— ip)} with <p subharmonic on the unit 
disc, ip = on a dense set of A and ip > and non identically zero, in particular it 
doesn't have Lipschitz boundary. 

3. Invariant Sets 

3.1. /—Invariant Relatively Compact Subsets. Let / : U — > C fe be a holo- 
morphic map with U CC C k or U Kobayashi hyperbolic. We assume that U is 
an open, connected and polynomially convex set. We say that a closed set C is 
/—invariant if /(C) C C. 

Proposition 3.1. LetC CC U be a closed f— invariant set, thenCp is f —invariant. 
Proof 

By hypothesis, C CC U. Choose z G Cp and suppose f(zo) Cp. Then there is a 
p.s.h. smooth function po in C fc , such that po < on Cp and Po(/(zo)) > 1- 
The function po ° / is p.s.h. on [/, po ° / < on C and po ° f is also p.s.h. on the 
holomorphic hull of C with respect to U, which is the same as Cp. It follows, by 
Maximum Principle, that po(/(^o)) < Oj which is a contradiction. □ 

Definition 3.2. A connected component il C U, of the set of points where {/ n }neN 
is equicontinuous, is recurrent if there exists po G ^ such that f ni (j>o) is relatively 
compact in 17 for some subsequence rii, i.e. if f2 contains a recurrent point pq. 

Proposition 3.3. If V — Int(C-p) ^ then the sequence {/"} n eN defined on V 
is a normal family and if V has a recurrent component U then there are three 
possibilities: 

i) / has an attractive periodic orbit, 

ii) there is a Siegel domain, i.e. there is U, a component ofV and a subsequence 
Hi, s.t. ffij -> Id, 

iii) if h is a limit of a subsequence of {f n } n eN, then < rank(h) < k. 
Proof 

We assume that for some po, f ni (po) — > p G U, and f ni converges uniformly on 
compact sets. We now write / n *+i _,1 < o f n% = / n *+i. Extracting a subsequence we 
get a limit h of /™i+i~"i such that h(p) = p, [7]. If /i is of rank 0, we show that 
p is an attractive fixed point, [7]. If /i is of maximal rank, then we get a Siegel 
domain, [7]. The theorem of Caratheodory-Cartan-Kaup-Wu, see [18] page 438 
and [13] page 66, describes the permitted eigenvalues. Otherwise for all possible h, 
< rank (h) < k. 

In [7] , Fornaess and Sibony prove a more precise result when / is an endomorphism 
of P . Their stronger result is valid only in dimension two. □ 

3.2. /—Invariant Non-Relatively Compact Subsets. 

Theorem 3.4. Let f : U — > C fc be a holomorphic open map defined on U, a bounded 
(or hyperbolic) open and connected subset ofC k . Assume that every point in dU 
has a p.s.h. barrier, i.e. if q G dU, there exists a p.s.h. function p q , p q < on 
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U, continuous such that lim p q (p) = 0. Suppose C is a f— invariant set in U. Let V 

bt the non-empty interior of C, where the adherence is with respect to U. We also 
assume that a connected component of V, W, contains a recurrent point po. Then 
there are three possibilities for W : 

1) it is the basin of attraction of an attracting periodic orbit; 

2) it is a Siegel domain; 

3) if h is a limit of a subsequence of {f n } n eN, on W, then < rank(h) < k. 
Proof 

We start proving that the sequence {f n } n &n is well defined on V. Since V C U 
is invariant, by continuity f(V) C U : indeed if p € V there exists a sequence of 
points p n G C such that p n — > p and hence f(p n ) f(p) = q <E U . We show 
now that f(V) C U. Suppose q G dU. Consider the barrier p q at q. The func- 
tion p q o f is p.s.h. and continuous on V, and p q o / < on V. But (p q o f)(p) = 
lim (pq°f)iPn) — um Pq(f(Pn)) = 0. Hence, by Maximum Principle, pg°/ = 0, 

n — >-\-oc n — > + oo 

i.e. f{V) G (p q = 0) G dU. This is impossible because / is open. Hence f(V) G U 
and f n (V) G C7, therefore the sequence {/™}neN is normal, since U is bounded. 
Now suppose that there exists a recurrent point po in W, a connected component 
of V. This means that there exists a sequence of n, — > +oo s.t. f ni (po) — > po G W 7 - 
We can always suppose that nj+i — rij — > +cxd. Taking a subsequence {i = we 
can suppose that the sequence {/™ i+1 ~ ni }i converges uniformly on compact sets of 
W to a holomorphic map h : W U s.t. /i(po) = Po- Indeed let = f ni {po)- 
Then /"^-"'Op.) = /" I+1 (po) = Pi+x- Hence /" 1+1 -" l (po) = p l+ i + 0{\ Pi -p \) so 
converges to po and therefore, necessarily, h(po) — po, [7]. 

Consider all maps h obtained in this way. If some h is of rank 0, then some iterate 
of / has po as an attractive fixed point and / has po as an attractive periodic point. 
If some h is of maximal rank k, then W is a Siegel domain, otherwise all the limit 
maps have lower rank r, < r < k. In [7] the authors analyze the case of holomor- 
phic endomorphisms of P 2 and thanks to the restriction to the dimension 2 and to 
the endomorphism case, the result there is much more precise: for example in case 
Hi), h(W) is always independent of h and attracts all orbits. □ 



Remark 3.5. If / is not open it is enough to assume that (p q = 0) does not contain 
the image of /. 

Corollary 3.6. Under the hypotheses of Theorem \3.4\ if C is an invariant closed 
set with a dense orbit in it or a countable union of closed invariant sets each one 
with a dense orbit, then the interior V of C doesn't contain recurrent points. 

Proof 

Indeed in the possible dynamical behaviours described in Theorem 13.41 when C is 
closed with a dense orbit cannot have interior; when we consider a countable union 
of closed sets with empty interior then, by Baire's theorem, the union of them is 
still with empty interior. □ 
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4. Forward invariant compact sets 

Theorem 4.1. Let U be a bounded star-shaped domain with respect to in C k 
and let U' be an open neighborhood of U. Let f : U' — * C fc , be a holomorphic map, 
/(0) = 0, / infective on U (i.e. f : U — > f(U) is a biholomorphic map) and f(U) 
is a Runge domain. Assume f(z) = Az + 0(z 2 ), with A a linear invertible map 
and with all the eigenvalues Xj, for 1 < j < k, of modulus 1. Then there exists a 
unique maximal connected compact set K, with £ K C U s.t. (K n dU) =/= and 
f{K) C K. Futhermore f is linearizable iff £ Lnt(K). 

Proof 

Define f^ n {z) = f(p n ■ z) with fi n e M, < fi n < 1 and ji n — > 1 for n — > +oo. Then 
"~ * / uniformly on 77 and \Jac(f^ n )(0)\ = \fi n \ ■ \Jac(A)\ < 1 because \/j, n \ < 1 
and | Jac(A)\ = 1; indeed f tln (z) — \i n • A ■ z + 0((fj, n ■ z) 2 ). 
For simplicity, we call /i :— fi n . 

Let fu'-—'U—>f(U) indeed fn(— ■ U) = f(U). Hence /„ is a biholomorphism 
H \i ^ 

from a star-shaped domain — • U to a Runge domain f(U) = fu(— ■ U). Now 

applying a result of Andersen-Lempert, [2] Theorem 2.1, to the biholomophism 

fn'-~-U—> f(U), we find a sequence of automorphisms g m of C , such that 

M 

<7m — * f fi for m — * +oo uniformly on compact subsets of U, i.e. the g m 's converge 
to / M , uniformly on compact sets and g m (0) = for all m. 
Since | Jac(/ M )(0)| < 1, then \Jac(g m )(0)\ < 1. 
Hence g m € ^4ut(C fe ) and g m :U — > ff m (J7) with OeC/n g m (U). 
Let B be a domain which is a homothetic of {/, i.e. = eJ7, sufficiently small s.t. 
g^ l 1 (B) C 77 i.e. £5 C (77 n<7 TO (?7)). Since the basin of attraction of for g m (i.e. 
(J g^ n (B)) is biholomorphic to C fe , [15], and in particular is unbounded, there 

nSN 

exists B„eff s.t. g^ no (B) C 17 but g~ (n ° +1) (B) <f. U (n > 1). 

We consider the one-parameter family {i?t}(>i where B t = t ■ B, 15J. Then we 

consider the t's for which: 

3 -™°(B t )cC7. 

The set is not empty because for t = 1 the inclusion is true. By continuity, there 
exists t s.t. 

g- n "(B I )cU 

and 

g- n °mn(dU)^t 

We call F m :=g^ n °{Bj). 

Then (F' m ) m gN is a sequence of compact sets in 77 s.t. gm(F m ) C -F m because 
g~ l n " +1 (Bj) C g m "° (Bj) : this follows from the description of the basin of attraction 
ofO. 

Each F m is a connected set because it is the closure of the pre-image by a biholo- 
morphism of a connected set. 

By compactness of the space K C {U) = {connected compact subsets of 77}, there 
exists a subsequence (mk)k£N t.c. F„ lk — > 7^ € fC c (U). Finally we prove that 

We use that: 
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(i) g m — > ffj, uniformly on compact subsets of U ; 

(ii) lim F mk = K^. 

k — >+oo 

Let x £ K^, then we want to prove that fp(x) £ K^. 
Since x £ K^, there exists a sequence x/c — » x with Xfe £ i 7 ^ by (ii). 
Then g mh {x k ) 6 F m(c and we can assume g mk ( x k) — ► 2/ G ifp, by (ii). 
But 5„ ifc -» / p for fc -> +oo by (i), so / M (x) = lim g mk (x k ) = y £ 

fc — >oo 

Hence Kn is /„— invariant. 

Therefore for each fx we have found a forward invariant connected compact set 
for /p and -K^ intersects dU. Now, with an argument similar to the one already 
used for {g m }meN and {F mk } k £fi, we prove that, up to considering a subsequence, 
Kfi n — > if in the Hausdorff metric. Since /^ n — > / uniformly on compact sets, we 
have that f(K) £ X and ii" touches dU. In order to have the unique, maximal, 
connected, invariant compact set, it is enough to take the closure of the union 
of all such compact sets K. Obviously, the closure of a union of /—invariant sets 
is still /—invariant and it is also connected because each compact set contains 0. 
Since intersects dU for all fj, n , also its limit K in the Hausdorff topology does. 
Suppose £ Int(K), we show that / is linearizable. The family (f n ) n &i ^ s locally 
equicontinuous on Int(K) and /(0) = 0. Following a standard trick, we define 

h(z):= Km ±Y,A-if*{z). 

The limit exists in a neighborhood of zero. Indeed there is a c > 1 such that 
/ n (B(0, r)) £ B(0,cr) £ K for all n. Then we can consider a limit map h for an 
appropriate subsequence rij. We have h(0) = 0, Jac(h)(0) = Id and we easily check 
that h(f) = Ah. □ 



Remark 4.2. If we take a sequence ji n > 1, fj, n — > 1, we can prove that there 
exists a maximal connected compact set invariant for / . In general the forward 
and backward invariant compact subsets are different, as the case of Henon maps 
shows, see Example 1 5 . 1 1 b elo w . 

Remark 4.3. We want to point out that K is not necessarily a proper subset of U , 
indeed if / is an automorphism of the ball M k C C k fixing 0, then K = B> k . 

Remark 4.4. Suppose that /, g are two commuting maps satisfying all the hy- 
potheses of Theorem 14. 11 then they share the same maximal, compact, connected, 
invariant set K 3 0. 

Indeed let Kf and K g be the maximal, compact, connected invariant sets con- 
taining 0, for / and g respectively, which exist by Theorem 14.11 Then consider 
/ o g(Kf) —go f(Kf) £ g(Kt), hence g(Kf) C Kf which implies that Kf £ K g . 
Analogously, considering g o f{K g ) = f o g(K g ), we can prove that K g £ Kf. 

5. Examples 

In this section we are going to prove that our theorem 14.11 is optimal, we mean 
that there exist a map / : B — > C fc which satisfy all the hypotheses of the Theorem 
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14.11 such that it has a forward invariant compact and connected set containing 
which touches the boundary of B but it doesn't admit a totally invariant compact 
and connected set containing which touches the boundary of B. 

Example 5.1. Let / be the following Henon map: 

f{z,w) = (z 2 + w,z). 

Then /(0,0) = (0,0) and 

^c(/)=(f J) 

So, at 0, Ai = 1, A 2 = -1, i.e. |Aj| = 1 for j = 1,2. Clearly / € Aut(C 2 ). From 
the well known study of the dynamics of /, there exist the following closed invariant 
subsets of C 2 : 

K+ = {zeC 2 | /"(» is bounded} 

Kj = {z G C 2 | f~ n {z) is bounded} 
and the following compact set of C 2 containing : 

K = Kjr\Kj. 

Consider a ball B(0, R) C C 2 with R » 1 such that 1(0, R) DD K. If we consider 
the restriction / : B(0, R) — > C 2 , by Theorem 14 . 1 1 there exists a connected compact 
subset X of B(0, R) which touches the dM(0,R), which is /—invariant and which 
contains 0. For any such X, we have X C Kj, [T7], because if z € X, f n (z) is 

bounded since X is /—invariant and compact. Hence X C (Kt n B(0, R)). It is 
well known from the study of the dynamics of Henon maps that: 

dist(f n (X),K) -> 

uniformly on compact sets. Hence there exists n € N such that dist(f n °(X), K) < 
— ■ dist(K, t?B(0, R)). So X cannot be at the same time forward and backward in- 
variant i.e. f(X) C X, but f(X) ^ X. 

If f no (X) is distant from K less than dist(dB(0, R),K), then it means that f n ° (X) C 
X and they are different. 

Hence, if we consider g := /"°, then g(X) CC X. 

Example 5.2. In some cases it is possible that the forward and the backward 
invariant compact sets coincide. For example, if in the previous example we consider 
a ball 1(0, r) which contains K = K+C\KJ and such that if n<9B(0, r) ^ 0, then the 
restriction of the Henon map / to B(0, r) admits a forward and backward invariant 
compact set K which touches the boundary of B(0,r). 

Remark 5.3. Let K be one of the /—invariant, connected and compact set of Theo- 
rem [4T] and let X = f] f n (K). The set X is connected because it is a decreasing 

intersection of connected sets, X 3 0, X is compact and f(X) = X. For example if 
/ is an Henon map, X — Kt n K 7. 
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